[Linux] Prevent GC from running during process teardown #57832
Merged
+5
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d-netto
force-pushed
the
dcn-kp-no-gc-on-teardown
branch
from
6be4221 to
71ba5fa
Compare
d-netto
force-pushed
the
dcn-kp-no-gc-on-teardown
branch
3 times, most recently
from
e1e3d4f to
4d64c18
Compare
vtjnash
reviewed
Mar 21, 2025
|
We should do this just before sending the signal to stop the thread though, otherwise we start running code concurrently with the GC. Can resume the GC immediately after though |
|
I'm confused, the signal handling thread doesn't touch the GC at all (I don't think it even gets adopted), and the exiting thread will run finalizers and such so its not gc safe |
d-netto
force-pushed
the
dcn-kp-no-gc-on-teardown
branch
from
4d64c18 to
2e2d508
Compare
|
Tested this, and the suggestion from here also fixes the false alarms' issue. |
d-netto
force-pushed
the
dcn-kp-no-gc-on-teardown
branch
from
2a2c495 to
5c9a592
Compare
vtjnash
reviewed
Mar 24, 2025
|
Seems like we don't call |
|
oh, good point, Windows' doesn't really have signals anyways so it is not relevant there |
3 tasks
d-netto
added
backport 1.12
KristofferC
pushed a commit
that referenced
this pull request
Mar 25, 2025
## Context We send a signal 15 to shutdown our servers. We noticed that some of our servers that receive the termination signal are segfaulting in GC, which leads to false alarms in our internal monitors that track GC-related crashes. ## Hypothesis We suspect this pathological case may be happening: - Process receives signal 15, which is captured by the signal listener thread. - Signal listener initiates process' teardown (e.g. through `raise`). - IIRC such operation is not atomic in Linux, i.e. the kernel will gradually kill the threads, but it's possible for us to spent a few ms in a state where part of the threads in the system are alive, and part have already been killed (this point needs some confirmation). - With part of the process alive, and part of the process dead, we try to enter a GC, see a bunch of Julia data structures in an intermediate/corrupted state, which leads us to crash when running the GC. ## Mitigation Since our main goal is to get rid of the GC crashes that happen around server shutdown, we believe that it would be sufficient to just prevent the last bullet point. I.e. we prevent the system from even running a GC when we're about to kill the process, and we wait for any ongoing GC to finish. Co-debugged with @kpamnany. (cherry picked from commit e1e3a46)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
We send a signal 15 to shutdown our servers.
We noticed that some of our servers that receive the termination signal are segfaulting in GC, which leads to false alarms in our internal monitors that track GC-related crashes.
Hypothesis
We suspect this pathological case may be happening:
raise).Mitigation
Since our main goal is to get rid of the GC crashes that happen around server shutdown, we believe that it would be sufficient to just prevent the last bullet point. I.e. we prevent the system from even running a GC when we're about to kill the process, and we wait for any ongoing GC to finish.
Co-debugged with @kpamnany.